Problem

Currently in sim2.cpp, when a simulated process dies, its DNS entry is immediately removed from the simulator's registry. While this works, it does not match real-world DNS caching/TTL behavior where stale records persist for a brief window. Because of this immediate cleanup, we miss out on simulating scenarios where clients attempt to connect to dead endpoints due to stale DNS information, which can hide bugs in our retry and connection recovery code.

Solution

We introduced a configurable delay to DNS record removal in the simulator:

1. Flow Knob: Added SIM_DNS_REMOVAL_MAX_DELAY in flow/Knobs. By default, this is 0.0 to preserve existing immediate-cleanup behavior. Under BUGGIFY, it is set to a random delay in the range of [0.0, 2.0] seconds.

2. Delayed Handler: Added delayedRemoveIp to SimRegisteredHandlerContext in HTTPServer.cpp. It logs a SimDNSDelayedRemoval trace event, fires a code probe, waits for the computed delay, and removes the registration.

3. Sim2 Integration: Replaced the immediate removal call at sim2.cpp:2228 with the new delayed approach when SIM_DNS_REMOVAL_MAX_DELAY > 0.

Co-author Attribution:

Special thanks to Renish Patel (@Renish-patel ) for the design approach, particularly suggesting the use of uncancellable() on the delayed future to prevent orphaned or leaked DNS entries if the parent actor is cancelled mid-wait.

Testing Done

• Staged and verified changes build successfully.
• Added trace events and code probes to verify simulation coverage and timing correctness.